YCBCR pulsed illumination scheme in a light deficient environment

ABSTRACT

The disclosure extends to methods, systems, and computer program products for producing an image in light deficient environments with luminance and chrominance emitted from a controlled light source.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/940,189, filed Jul. 27, 2020, which is a continuation of U.S.application Ser. No. 16/398,114, filed Apr. 29, 2019, which is acontinuation of U.S. application Ser. No. 15/701,264, filed on Sep. 11,2017 (now U.S. Pat. No. 10,277,875), which is a continuation of U.S.application Ser. No. 15/369,170, filed on Dec. 5, 2016 (now U.S. Pat.No. 9,762,879, issued Sep. 12, 2017), which is a division of U.S.application Ser. No. 13/952,570, filed on Jul. 26, 2013 (now U.S. Pat.No. 9,516,239, issued Dec. 6, 2016) and claims the benefit of U.S.Provisional Patent Application No. 61/676,289, filed on Jul. 26, 2012,and U.S. Provisional Patent Application No. 61/790,487, filed on Mar.15, 2013, and U.S. Provisional Patent Application No. 61/790,719, filedon Mar. 15, 2013 and U.S. Provisional Patent Application No. 61/791,473,filed on Mar. 15, 2013, which are hereby incorporated by referenceherein in their entireties, including but not limited to those portionsthat specifically appear hereinafter, the incorporation by referencebeing made with the following exception: In the event that any portionof the above-referenced applications is inconsistent with thisapplication, this application supersedes said above-referencedapplications.

BACKGROUND

Advances in technology have provided advances in imaging capabilitiesfor medical use. One area that has enjoyed some of the most beneficialadvances is that of endoscopic surgical procedures because of theadvances in the components that make up an endoscope.

The disclosure relates generally to electromagnetic sensing and sensorsin relation to creating a video stream having chrominance and luminancepulses from a controlled light source. The features and advantages ofthe disclosure will be set forth in the description which follows, andin part will be apparent from the description, or may be learned by thepractice of the disclosure without undue experimentation. The featuresand advantages of the disclosure may be realized and obtained by meansof the instruments and combinations particularly pointed out herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive implementations of the disclosure aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified. Advantages of the disclosure will becomebetter understood with regard to the following description andaccompanying drawings.

FIG. 1 illustrates a graphical representation of the operation of apixel array in accordance with the principles and teachings of thedisclosure;

FIG. 2 illustrates a graphical representation of a pixel array for aplurality of frames in accordance with the principles and teachings ofthe disclosure;

FIG. 3A illustrates a schematic of an embodiment of an operationsequence of chrominance and luminance frames in accordance with theprinciples and teachings of the disclosure;

FIG. 3B illustrates a schematic of an embodiment of an operationsequence of chrominance and luminance frames in accordance with theprinciples and teachings of the disclosure;

FIG. 3C illustrates a schematic of an embodiment of an operationsequence of chrominance and luminance frames in accordance with theprinciples and teachings of the disclosure;

FIG. 4 illustrates an embodiment of sensor and emitter modulation inaccordance with the principles and teachings of the disclosure;

FIG. 5 illustrates an embodiment of sensor and emitter patterns inaccordance with the principles and teachings of the disclosure;

FIG. 6A illustrates an embodiment of sensor and emitter patterns inaccordance with the principles and teachings of the disclosure;

FIG. 6B illustrates an embodiment of sensor and emitter patterns inaccordance with the principles and teachings of the disclosure;

FIG. 7 illustrates a graphical representation of the operation of apixel array having pixels of differing pixel sensitivities in accordancewith the principles and teachings of the disclosure;

FIG. 8 illustrates a graphical representation of the operation of apixel array having pixels of differing pixel sensitivities in accordancewith the principles and teachings of the disclosure;

FIG. 9 illustrates a flow chart of the operation of a pixel array inaccordance with the principles and teachings of the disclosure;

FIG. 10 illustrates a flow chart of the operation of a pixel array inaccordance with the principles and teachings of the disclosure;

FIG. 11 illustrates a flow chart of the operation of a pixel array inaccordance with the principles and teachings of the disclosure;

FIG. 12A illustrates a graphical representation of the operation of apixel array in accordance with the principles and teachings of thedisclosure;

FIG. 12B illustrates a graphical representation of the operation of apixel array in accordance with the principles and teachings of thedisclosure;

FIG. 13 illustrates an embodiment of supporting hardware in accordancewith the principles and teachings of the disclosure;

FIGS. 14A and 14B illustrate an implementation having a plurality ofpixel arrays for producing a three dimensional image in accordance withthe teachings and principles of the disclosure;

FIGS. 15A and 15B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor built on aplurality of substrates, wherein a plurality of pixel columns formingthe pixel array are located on the first substrate and a plurality ofcircuit columns are located on a second substrate and showing anelectrical connection and communication between one column of pixels toits associated or corresponding column of circuitry; and

FIGS. 16A and 16B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor having aplurality of pixel arrays for producing a three dimensional image,wherein the plurality of pixel arrays and the image sensor are built ona plurality of substrates.

DETAILED DESCRIPTION

The disclosure extends to methods, systems, and computer based productsfor digital imaging that may be primarily suited to medicalapplications. In the following description of the disclosure, referenceis made to the accompanying drawings, which form a part hereof, and inwhich is shown by way of illustration specific implementations in whichthe disclosure may be practiced. It is understood that otherimplementations may be utilized and structural changes may be madewithout departing from the scope of the disclosure.

Luminance-chrominance based color spaces date back to the advent ofcolor television, when color image transmission was required to becompatible with older monochrome CRTs. The luminance componentcorresponds to the (color-agnostic) brightness aspect of the image data.The color information is carried in the remaining two channels. Theseparation of image data into the luminance and chrominance componentsis still an important process in modern digital imaging systems, sinceit is closely related to the human visual system.

The human retina contains arrays of two basic photoreceptor cell types;rods and cones. The rods provide the brightness information and haveabout a factor-20 greater overall spatial density than the cones. Thecones are much less sensitive and there are three basic types, havingpeak responses at three different wavelengths. The spectral response ofthe rods, which peaks in the green region, is the basis for computingluminance color-space conversion coefficients. Since rods have thegreater density, the spatial resolution of an image representation ismuch more important for the luminance component than for eitherchrominance component. Camera designers and image processing engineersseek to account for this fact in several ways, e.g., by spatiallyfiltering the chrominance channels to reduce noise and by affordinggreater relative system bandwidth to luminance data.

In describing the subject matter of the disclosure, the followingterminology will be used in accordance with the definitions set outbelow.

It must be noted that, as used in this specification, the singular forms“a,” “an,” and “the” include plural referents unless the context clearlydictates otherwise.

As used herein, the terms “comprising,” “including,” “containing,”“characterized by,” and grammatical equivalents thereof are inclusive oropen-ended terms that do not exclude additional, unrecited elements ormethod steps.

As used herein, the phrase “consisting of” and grammatical equivalentsthereof exclude any element or step not specified.

As used herein, the phrase “consisting essentially of” and grammaticalequivalents thereof limit the scope of a claim, if any, to the specifiedmaterials or steps and those that do not materially affect the basic andnovel characteristic or characteristics of the claimed disclosure.

As used herein, the term “proximal” shall refer broadly to the conceptof a portion nearest an origin.

As used herein, the term “distal” shall generally refer to the oppositeof proximal, and thus to the concept of a portion farther from anorigin, or a furthest portion, depending upon the context.

Referring now to the figures, FIG. 1 illustrates the basic timing ofsingle frame capture by a conventional CMOS sensor. Co-pending U.S.patent application Ser. No. 13/952,518 entitled CONTINUOUS VIDEO IN ALIGHT DEFICIENT ENVIRONMENT is hereby incorporated by this referenceinto this disclosure as if fully set forth herein. It will beappreciated that the x direction corresponds to time and the diagonallines indicate the activity of an internal pointer that reads out eachframe of data, one line at time. The same pointer is responsible forresetting each row of pixels for the next exposure period. The netintegration time for each row is equivalent, but they are staggered intime with respect to one another due to the rolling reset and readprocess. Therefore, for any scenario in which adjacent frames arerequired to represent different constitutions of light, the only optionfor having each row be consistent is to pulse the light between thereadout cycles. More specifically, the maximum available periodcorresponds to the sum of the blanking time plus any time during whichoptical black or optically blind (OB) rows are serviced at the start orend of the frame.

An example illumination sequence is a repeating pattern of four frames(R-G-B-G). As for the Bayer pattern of color filters, this provides forgreater luminance detail than chrominance. This approach is accomplishedby strobing the scene with either laser or light-emitting diodes at highspeed, under the control of the camera system, and by virtue of aspecially designed CMOS sensor with high speed readout. The principalbenefit is that the sensor can accomplish the same spatial resolutionwith significantly fewer pixels compared with conventional Bayer or3-sensor cameras. Therefore, the physical space occupied by the pixelarray may be reduced. The actual pulse periods may differ within therepeating pattern, as illustrated in FIG. 2 . This is useful for, e.g.,apportioning greater time to the components that require the greaterlight energy or those having the weaker sources. As long as the averagecaptured frame rate is an integer multiple of the requisite final systemframe rate, the data may simply be buffered in the signal processingchain as appropriate.

The facility to reduce the CMOS sensor chip-area to the extent allowedby combining all of these methods is particularly attractive for smalldiameter (˜3-10 mm) endoscopy. In particular, it allows for endoscopedesigns in which the sensor is located in the space-constrained distalend, thereby greatly reducing the complexity and cost of the opticalsection, while providing high definition video. A consequence of thisapproach is that to reconstruct each final, full color image, requiresthat data be fused from three separate snapshots in time. Any motionwithin the scene, relative to the optical frame of reference of theendoscope, will generally degrade the perceived resolution, since theedges of objects appear at slightly different locations within eachcaptured component. In this disclosure, a means of diminishing thisissue is described which exploits the fact that spatial resolution ismuch more important for luminance information, than for chrominance.

The basis of the approach is that, instead of firing monochromatic lightduring each frame, combinations of the three wavelengths are used toprovide all of the luminance information within a single image. Thechrominance information is derived from separate frames with, e.g., arepeating pattern such as Y-Cb-Y-Cr. While it is possible to providepure luminance data by a shrewd choice of pulse ratios, the same is nottrue of chrominance. However, a workaround for this is presented in thisdisclosure.

In an embodiment, as illustrated in FIG. 3A, an endoscopic system 300 amay comprise a pixel array 302 a having uniform pixels and the system300 a may be operated to receive Y (luminance pulse) 304 a, Cb(ChromaBlue) 306 a and Cr (ChromaRed) 308 a pulses.

In an embodiment, as illustrated in FIG. 3B, an endoscopic system 300 bmay comprise a pixel array 302 b having uniform pixels and the systemmay be operated to receive Y (luminance pulse) 304 b, λY+Cb (ModulatedChromaBlue) 306 b and δY+Cr (Modulated ChromaRed) 308 b pulses.

In an embodiment, as illustrated in FIG. 3C, an endoscopic system 300 cmay comprise a pixel array 302 c having checker patterned (alternating)pixels and the system may be operated to receive Y (luminance pulse) 304c, λY+Cb (Modulated ChromaBlue) 306 c and δY+Cr (Modulated ChromaRed)308 c pulses. Within the luminance frames, the two exposure periods areapplied for the purpose of extending the dynamic range (YL and YS,corresponding to the long and short exposures).

FIG. 4 illustrates the general timing relationship within a 4-framecycle, between pulsed mixtures of three wavelengths and the readoutcycle of a monochrome CMOS sensor.

Essentially there are three monochromatic pulsed light sources under thefast control of the camera and a special design of monochromatic CMOSimage sensor which enables high final progressive video rates of 60 Hzor more. Periodic sequences of monochromatic red, green and blue framesare captured, e.g., with an R-G-B-G pattern, and assembled into sRGBimages in the image signal processor chain (ISP). The light-pulse andsensor readout timing relationship is shown in FIG. 5 . In order toprovide pure luminance information in the same frame, all three sourcesare pulsed in unison with light energies that are modulated according tothe color transformation coefficients that convert from RGB space toYCbCr (as per the ITU-R BT.709 HD standard):

$\begin{bmatrix}Y \\{Cb} \\{Cr}\end{bmatrix} = {\begin{bmatrix}R \\G \\B\end{bmatrix}\begin{bmatrix}0.183 & 0.614 & {{0.0}62} \\{- 0.101} & {{- {0.3}}39} & {{0.4}39} \\{{0.4}39} & {{- {0.3}}99} & {{- {0.0}}40}\end{bmatrix}}$

It will be appreciated that other color space conversion standards maybe implemented by the disclosure, including but not limited to, ITU-RBT.709 HD standard, ITU-R BT.601 standard, and ITU-R BT.2020 standard.

If white balance is being performed in the illumination domain, thenthis modulation is imposed in addition to the white balance modulation.

To complete a full color image requires that the two components ofchrominance also be provided. However, the same algorithm that wasapplied for luminance cannot be directly applied for chrominance imagessince it is signed, as reflected in the fact that some of the RGBcoefficients are negative. The solution to this is to add a degree ofluminance of sufficient magnitude that all of the final pulse energiesbecome positive. As long as the color fusion process in the ISP is awareof the composition of the chrominance frames, they can be decoded bysubtracting the appropriate amount of luminance from a neighboringframe. The pulse energy proportions are given by:

Y = 0.183 ⋅ R + 0.614 ⋅ G + 0.062 ⋅ BCb = λ ⋅ Y − 0.101 ⋅ R − 0.339 ⋅ G + 0.439 ⋅ BCr = δ ⋅ Y + 0.439 ⋅ R − 0.399 ⋅ G − 0.040 ⋅ B where${\lambda \geq \frac{{0.3}39}{{0.6}14}} = {{0.5}52}$${\delta \geq \frac{{0.3}99}{{0.6}14}} = {{0.6}50}$

The timing for the general case is shown in FIG. 6A. It turns out thatif the λ factor is equal to 0.552; both the red and the green componentsare exactly cancelled, in which case the Cb information can be providedwith pure blue light. Similarly, setting δ=0.650 cancels out the blueand green components for Cr which becomes pure red. This particularexample is illustrated in FIG. 6B, which also depicts λ and δ as integermultiples of ½⁸. This is a convenient approximation for the digitalframe reconstruction (see later discussion).

Referring now to FIG. 7 , there is illustrated a general timing diagramfor this process. The exposure periods for the two flavors of pixel arecontrolled by two internal signals within the image sensor, depicted asTX1 and TX2 in the figure. In fact, it is possible to do this at thesame time as extending the dynamic range for the luminance frame, whereit is most needed, since the two integration times can be adjusted on aframe by frame basis (see FIGS. 3 a-3 c ). The benefit is that the colormotion artifacts are less of an issue if all the data is derived fromtwo frames versus three. There is of course a subsequent loss of spatialresolution for the chrominance data but that is of negligibleconsequence to the image quality for the reasons discussed earlier.

An inherent property of the monochrome wide dynamic range array is thatthe pixels that have the long integration time must integrate a supersetof the light seen by the short integration time pixels. Co-pending U.S.patent application Ser. No. 13/952,564 entitled WIDE DYNAMIC RANGE USINGMONOCHROMATIC SENSOR is hereby incorporated by this reference into thisdisclosure as if fully set forth herein. For regular wide dynamic rangeoperation in the luminance frames, that is desirable. For thechrominance frames it means that the pulsing must be controlled inconjunction with the exposure periods so as to provide, e.g., λY+Cb fromthe start of the long exposure and switch to δY+Cr at the point that theshort pixels are turned on (both pixel types have their chargestransferred at the same time). During color fusion, this would beaccounted for. FIG. 8 shows the specific timing diagram for thissolution.

A typical ISP involves first taking care of any necessary sensor andoptical corrections (such as defective pixel elimination, lens shadingetc.), then in turn; white balance, demosaic/color fusion and colorcorrection.

Before finally applying gamma to place the data in the standard sRGBspace, there might typically be some operations (e.g., edge enhancement)and/or adjustments (e.g., saturation) performed in an alternative colorspace such as YCbCr or HSL. FIG. 9 depicts a basic ISP core that wouldbe appropriate for the R-G-B-G pulsing scheme. In this example, the datais converted to YCbCr in order to apply edge enhancement in theluminance plane and conduct filtering of the chrominance, then convertedback to linear RGB.

In the case of the Y-Cb-Y-Cr pulsing scheme, the image data is alreadyin the YCbCr space following the color fusion. Therefore, in this caseit makes sense to perform luminance and chrominance based operations upfront, before converting back to linear RGB to perform the colorcorrection etc. See FIG. 10 .

The color fusion process is more straightforward than de-mosaic, whichis necessitated by the Bayer pattern, since there is no spatialinterpolation. It does require buffering of frames though in order tohave all of the necessary information available for each pixel, asindicated in FIG. 11 . FIG. 12A shows the general situation ofpipelining of data for the Y-Cb-Y-Cr pattern which yields 1 full colorimage per two raw captured images. This is accomplished by using eachchrominance sample twice. In FIG. 12B the specific example of a 120 Hzframe capture rate providing 60 Hz final video is drawn.

The linear Y, Cb and Cr components for each pixel may be computed thus:

Y_(i) = 2^(m − 4) + (x_(i, n − 1) − K) ${\begin{Bmatrix}{{Cb}_{i} = {2^{m - 1} + ( {x_{i,n} - K} ) - {\lambda \cdot ( {x_{i,{n - 1}} - K} )}}} \\{{Cr}_{i} = {2^{m - 1} + ( {x_{i,{n - 2}} - K} ) - {\delta \cdot ( {x_{i,{n - 1}} - K} )}}}\end{Bmatrix}\mspace{14mu}{when}\mspace{14mu} n} = {{‘{Cb}’}\mspace{14mu}{frame}}$${\begin{Bmatrix}{{Cb}_{i} = {2^{m - 1} + ( {x_{i,{n - 2}} - K} ) - {\lambda \cdot ( {x_{i,{n - 1}} - K} )}}} \\{{Cr}_{i} = {2^{m - 1} + ( {x_{i,n} - K} ) - {\delta \cdot ( {x_{i,{n - 1}} - K} )}}}\end{Bmatrix}\mspace{14mu}{when}\mspace{14mu} n} = {{‘{Cr}’}\mspace{14mu}{frame}}$

Where x_(i,n) is the input data for pixel i in frame n, m is thepipeline bit-width of the ISP and K is the ISP black offset level at theinput to the color fusion block, (if applicable). Since chrominance issigned it is conventionally centered at 50% of the digital dynamic range(2^(m-1)).

If two exposures are used to provide both chrominance components in thesame frame as described earlier, the two flavors of pixel are separatedinto two buffers. The empty pixels are then filled in using, e.g.,linear interpolation. At this point, one buffer contains a full image ofδY+Cr data and the other; δY+Cr+λY+Cb. The δY+Cr buffer is subtractedfrom the second buffer to give λY+Cb. Then the appropriate proportion ofluminance data from the Y frames is subtracted out for each.

Implementations of the disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Implementations within the scope of thedisclosure may also include physical and other computer-readable mediafor carrying or storing computer-executable instructions and/or datastructures. Such computer-readable media can be any available media thatcan be accessed by a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arecomputer storage media (devices). Computer-readable media that carrycomputer-executable instructions are transmission media. Thus, by way ofexample, and not limitation, implementations of the disclosure cancomprise at least two distinctly different kinds of computer-readablemedia: computer storage media (devices) and transmission media.

Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM,solid state drives (“SSDs”) (e.g., based on RAM), Flash memory,phase-change memory (“PCM”), other types of memory, other optical diskstorage, magnetic disk storage or other magnetic storage devices, or anyother medium which can be used to store desired program code means inthe form of computer-executable instructions or data structures andwhich can be accessed by a general purpose or special purpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. In an implementation, a sensor andcamera control unit may be networked in order to communicate with eachother, and other components, connected over the network to which theyare connected. When information is transferred or provided over anetwork or another communications connection (either hardwired,wireless, or a combination of hardwired or wireless) to a computer, thecomputer properly views the connection as a transmission medium.Transmissions media can include a network and/or data links which can beused to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

As can be seen in FIG. 13 , various computer system components, programcode means in the form of computer-executable instructions or datastructures that can be transferred automatically from transmission mediato computer storage media (devices) (or vice versa). For example,computer-executable instructions or data structures received over anetwork or data link can be buffered in RAM within a network interfacemodule (e.g., a “NIC”), and then eventually transferred to computersystem RAM and/or to less volatile computer storage media (devices) at acomputer system. RAM can also include solid state drives (SSDs or PCIxbased real time memory tiered Storage, such as FusionIO). Thus, itshould be understood that computer storage media (devices) can beincluded in computer system components that also (or even primarily)utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. The computerexecutable instructions may be, for example, binaries, intermediateformat instructions such as assembly language, or even source code.Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined herein is not necessarily limited to thedescribed features or acts described above. Rather, the describedfeatures and acts are disclosed as examples.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, control units, camera controlunits, hand-held devices, hand pieces, multi-processor systems,microprocessor-based or programmable consumer electronics, network PCs,minicomputers, mainframe computers, mobile telephones, PDAs, tablets,pagers, routers, switches, various storage devices, and the like. Itshould be noted that any of the above mentioned computing devices may beprovided by or located within a brick and mortar location. Thedisclosure may also be practiced in distributed system environmentswhere local and remote computer systems, which are linked (either byhardwired data links, wireless data links, or by a combination ofhardwired and wireless data links) through a network, both performtasks. In a distributed system environment, program modules may belocated in both local and remote memory storage devices.

Further, where appropriate, functions described herein can be performedin one or more of: hardware, software, firmware, digital components, oranalog components. For example, one or more application specificintegrated circuits (ASICs) or field programmable gate arrays can beprogrammed to carry out one or more of the systems and proceduresdescribed herein. Certain terms are used throughout the followingdescription to refer to particular system components. As one skilled inthe art will appreciate, components may be referred to by differentnames. This document does not intend to distinguish between componentsthat differ in name, but not function.

FIG. 13 is a block diagram illustrating an example computing device 100.Computing device 100 may be used to perform various procedures, such asthose discussed herein. Computing device 100 can function as a server, aclient, or any other computing entity. Computing device can performvarious monitoring functions as discussed herein, and can execute one ormore application programs, such as the application programs describedherein. Computing device 100 can be any of a wide variety of computingdevices, such as a desktop computer, a notebook computer, a servercomputer, a handheld computer, camera control unit, tablet computer andthe like.

Computing device 100 includes one or more processor(s) 102, one or morememory device(s) 104, one or more interface(s) 106, one or more massstorage device(s) 108, one or more Input/Output (I/O) device(s) 110, anda display device 130 all of which are coupled to a bus 112. Processor(s)102 include one or more processors or controllers that executeinstructions stored in memory device(s) 104 and/or mass storagedevice(s) 108. Processor(s) 102 may also include various types ofcomputer-readable media, such as cache memory.

Memory device(s) 104 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM) 114) and/ornonvolatile memory (e.g., read-only memory (ROM) 116). Memory device(s)104 may also include rewritable ROM, such as Flash memory.

Mass storage device(s) 108 include various computer readable media, suchas magnetic tapes, magnetic disks, optical disks, solid-state memory(e.g., Flash memory), and so forth. As shown in FIG. 13 , a particularmass storage device is a hard disk drive 124. Various drives may also beincluded in mass storage device(s) 108 to enable reading from and/orwriting to the various computer readable media. Mass storage device(s)108 include removable media 126 and/or non-removable media.

I/O device(s) 110 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 100.Example I/O device(s) 110 include digital imaging devices,electromagnetic sensors and emitters, cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, lenses, CCDs or other imagecapture devices, and the like.

Display device 130 includes any type of device capable of displayinginformation to one or more users of computing device 100. Examples ofdisplay device 130 include a monitor, display terminal, video projectiondevice, and the like.

Interface(s) 106 include various interfaces that allow computing device100 to interact with other systems, devices, or computing environments.Example interface(s) 106 may include any number of different networkinterfaces 120, such as interfaces to local area networks (LANs), widearea networks (WANs), wireless networks, and the Internet. Otherinterface(s) include user interface 118 and peripheral device interface122. The interface(s) 106 may also include one or more user interfaceelements 118. The interface(s) 106 may also include one or moreperipheral interfaces such as interfaces for printers, pointing devices(mice, track pad, etc.), keyboards, and the like.

Bus 112 allows processor(s) 102, memory device(s) 104, interface(s) 106,mass storage device(s) 108, and I/O device(s) 110 to communicate withone another, as well as other devices or components coupled to bus 112.Bus 112 represents one or more of several types of bus structures, suchas a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

For purposes of illustration, programs and other executable programcomponents are shown herein as discrete blocks, although it isunderstood that such programs and components may reside at various timesin different storage components of computing device 100, and areexecuted by processor(s) 102. Alternatively, the systems and proceduresdescribed herein can be implemented in hardware, or a combination ofhardware, software, and/or firmware. For example, one or moreapplication specific integrated circuits (ASICs) can be programmed tocarry out one or more of the systems and procedures described herein.

FIGS. 14A and 14B illustrate a perspective view and a side view,respectively, of an implementation of a monolithic sensor 2900 having aplurality of pixel arrays for producing a three dimensional image inaccordance with the teachings and principles of the disclosure. Such animplementation may be desirable for three dimensional image capture,wherein the two pixel arrays 2902 and 2904 may be offset during use. Inanother implementation, a first pixel array 2902 and a second pixelarray 2904 may be dedicated to receiving a predetermined range of wavelengths of electromagnetic radiation, wherein the first pixel array isdedicated to a different range of wave length electromagnetic radiationthan the second pixel array.

FIGS. 15A and 15B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor 3000 built on aplurality of substrates. As illustrated, a plurality of pixel columns3004 forming the pixel array are located on the first substrate 3002 anda plurality of circuit columns 3008 are located on a second substrate3006. Also illustrated in the figure are the electrical connection andcommunication between one column of pixels to its associated orcorresponding column of circuitry. In one implementation, an imagesensor, which might otherwise be manufactured with its pixel array andsupporting circuitry on a single, monolithic substrate/chip, may havethe pixel array separated from all or a majority of the supportingcircuitry. The disclosure may use at least two substrates/chips, whichwill be stacked together using three-dimensional stacking technology.The first 3002 of the two substrates/chips may be processed using animage CMOS process. The first substrate/chip 3002 may be comprisedeither of a pixel array exclusively or a pixel array surrounded bylimited circuitry. The second or subsequent substrate/chip 3006 may beprocessed using any process, and does not have to be from an image CMOSprocess. The second substrate/chip 3006 may be, but is not limited to, ahighly dense digital process in order to integrate a variety and numberof functions in a very limited space or area on the substrate/chip, or amixed-mode or analog process in order to integrate for example preciseanalog functions, or a RF process in order to implement wirelesscapability, or MEMS (Micro-Electro-Mechanical Systems) in order tointegrate MEMS devices. The image CMOS substrate/chip 3002 may bestacked with the second or subsequent substrate/chip 3006 using anythree-dimensional technique. The second substrate/chip 3006 may supportmost, or a majority, of the circuitry that would have otherwise beenimplemented in the first image CMOS chip 3002 (if implemented on amonolithic substrate/chip) as peripheral circuits and therefore haveincreased the overall system area while keeping the pixel array sizeconstant and optimized to the fullest extent possible. The electricalconnection between the two substrates/chips may be done throughinterconnects 3003 and 3005, which may be wirebonds, bump and/or TSV(Through Silicon Via).

FIGS. 16A and 16B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor 3100 having aplurality of pixel arrays for producing a three dimensional image. Thethree dimensional image sensor may be built on a plurality of substratesand may comprise the plurality of pixel arrays and other associatedcircuitry, wherein a plurality of pixel columns 3104 a forming the firstpixel array and a plurality of pixel columns 3104 b forming a secondpixel array are located on respective substrates 3102 a and 3102 b,respectively, and a plurality of circuit columns 3108 a and 3108 b arelocated on a separate substrate 3106. Also illustrated are theelectrical connections and communications between columns of pixels toassociated or corresponding column of circuitry.

It will be appreciated that the teachings and principles of thedisclosure may be used in a reusable device platform, a limited usedevice platform, a re-posable use device platform, or asingle-use/disposable device platform without departing from the scopeof the disclosure. It will be appreciated that in a re-usable deviceplatform an end-user is responsible for cleaning and sterilization ofthe device. In a limited use device platform the device can be used forsome specified amount of times before becoming inoperable. Typical newdevice is delivered sterile with additional uses requiring the end-userto clean and sterilize before additional uses. In a re-posable usedevice platform a third-party may reprocess the device (e.g., cleans,packages and sterilizes) a single-use device for additional uses at alower cost than a new unit. In a single-use/disposable device platform adevice is provided sterile to the operating room and used only oncebefore being disposed of.

Additionally, the teachings and principles of the disclosure may includeany and all wavelengths of electromagnetic energy, including the visibleand non-visible spectrums, such as infrared (IR), ultraviolet (UV), andX-ray.

In an embodiment, a method for digital imaging for use with an endoscopein ambient light deficient environments may comprise: actuating anemitter to emit a plurality of pulses of electromagnetic radiation tocause illumination within the light deficient environment, wherein saidpulses comprise a first pulse that is within a first wavelength rangethat comprises a first portion of electromagnetic spectrum, wherein saidpulses comprise a second pulse that is within a second wavelength rangethat comprises a second portion of electromagnetic spectrum, whereinsaid pulses comprise a third pulse that is with is a third wavelengthrange that comprises a third portion of electromagnetic spectrum;pulsing said plurality of pulses at a predetermined interval; sensingreflected electromagnetic radiation from said pulses with a pixel arrayto create a plurality of image frames, wherein said pixel array is readat an interval that corresponds to the pulse interval of said laseremitter; and creating a stream of images by combining the plurality ofimage frames to form a video stream. In an embodiment, said first pulsecomprises chrominance red. In an embodiment, said second pulse compriseschrominance blue. In an embodiment, said third pulse comprises aluminance pulse. In an embodiment, said luminance pulse is created bypulsing a red pulse and a blue pulse and a green pulse. In such anembodiment, said red pulse is modulated relative to the blue and greenpulse such that the red pulse has a positive chrominance value. In anembodiment, said blue pulse is modulated relative to the red and greenpulse such that the blue pulse has a positive chrominance value. In anembodiment, said green pulse is modulated relative to the blue and redpulse such that the green pulse has a positive chrominance value. In anembodiment, the method further comprises modulating the plurality ofpulses by a value such that the chrominance value of each pulse ispositive. In an embodiment, the method further comprises removing pulsemodulation values from during image stream construction. In such anembodiment, the process of modulating comprises adding a luminance valueto the plurality of pulses. In an embodiment, the luminance value formodulation is an integer that is a multiple of (½)⁸. In an embodiment, aluminance value for modulation of 0.552 cancels out red chrominance andgreen chrominance. In an embodiment, a luminance value for modulation of0.650 cancels out blue chrominance and green chrominance. In anembodiment, the method further comprises reducing noise while creatingthe stream of image frames. In an embodiment, the method furthercomprises adjusting white balance while creating the stream of mageframes. In an embodiment, said third pulse is a luminance pulse that ispulses twice as often as the first and second pulses. In an embodiment,said luminance pulse is sensed by long exposure pixel and short exposurepixels within a pixel array. In an embodiment, the method furthercomprises sensing data generated by a plurality of pixel arrays andcombining said data into a three dimensional image stream.

It will be appreciated that various features disclosed herein providesignificant advantages and advancements in the art. The followingembodiments are exemplary of some of those features.

In the foregoing Detailed Description of the Disclosure, variousfeatures of the disclosure are grouped together in a single embodimentfor the purpose of streamlining the disclosure. This method ofdisclosure is not to be interpreted as reflecting an intention that thedisclosure requires more features than are expressly recited in eachclaim, if any. Rather, inventive aspects lie in less than all featuresof a single foregoing disclosed embodiment.

It is to be understood that the above-described arrangements are onlyillustrative of the application of the principles of the disclosure.Numerous modifications and alternative arrangements may be devised bythose skilled in the art without departing from the spirit and scope ofthe disclosure.

Thus, while the disclosure has been shown in the drawings and describedabove with particularity and detail, it will be apparent to those ofordinary skill in the art that numerous modifications, including, butnot limited to, variations in size, materials, shape, form, function andmanner of operation, assembly and use may be made without departing fromthe principles and concepts set forth herein.

Further, where appropriate, functions described herein can be performedin one or more of: hardware, software, firmware, digital components, oranalog components. For example, one or more application specificintegrated circuits (ASICs) can be programmed to carry out one or moreof the systems and procedures described herein. Certain terms are usedthroughout the following description to refer to particular systemcomponents. As one skilled in the art will appreciate, components may bereferred to by different names. This document does not intend todistinguish between components that differ in name, but not function.

What is claimed is:
 1. A system for digital imaging in an ambient lightdeficient environment comprising: an emitter configured to emit aplurality of pulses of electromagnetic radiation; an imaging sensorcomprising an array of pixels for sensing electromagnetic radiation; anda control unit comprising a processor and wherein the control unit is inelectrical communication with the imaging sensor and the emitter;wherein the control unit is configured to synchronize the emitter andthe imaging sensor to produce a plurality of image reference frames;wherein the imaging sensor generates image data from sensedelectromagnetic radiation that produces the plurality of image referenceframes comprising: a first luminance frame comprising first luminanceimage data; a second luminance frame comprising second luminance imagedata; a first chrominance frame comprising first chrominance data; asecond chrominance frame comprising second chrominance data; and a thirdchrominance frame comprising third chrominance data; wherein the firstluminance frame, the first chrominance frame, and the second chrominanceframe are combined to form a first color image frame; wherein the secondchrominance frame is reused such that the second chrominance frame iscombined with the third chrominance frame and the second luminance frameto form a second color image frame.
 2. The system of claim 1, whereinthe plurality of image reference frames comprises a plurality ofchrominance frames and a plurality of luminance frames; and wherein theplurality of chrominance frames in the plurality of image referenceframes alternate between blue chrominance frames having blue chrominancedata and red chrominance frames having red chrominance data.
 3. Thesystem of claim 1, wherein the first chrominance frame has bluechrominance data, the second chrominance frame has red chrominance data,and the third chrominance frame has blue chrominance data; or whereinthe first chrominance frame has red chrominance data, wherein the secondchrominance frame has blue chrominance data, and the third chrominanceframe has red chrominance data.
 4. The system of claim 1, wherein theplurality of image reference frames further comprises: a third luminanceframe comprising third luminance image data; and a fourth chrominanceframe comprising fourth chrominance data; wherein the third chrominanceframe is combined with the fourth chrominance frame and a thirdluminance frame to form a third color image frame.
 5. The system ofclaim 4, wherein the first chrominance frame has blue chrominance data,the second chrominance frame has red chrominance data, the thirdchrominance frame has blue chrominance data, and the fourth chrominanceframe has red chrominance data; or wherein the first chrominance framehas red chrominance data, the second chrominance frame has bluechrominance data, the third chrominance frame has red chrominance data,and the fourth chrominance frame has blue chrominance data.
 6. Thesystem of claim 2, wherein luminance frames are represented in theplurality of image reference frames twice as often as red chrominanceframes and twice as often as blue chrominance frames.
 7. The system ofclaim 2, wherein each of the plurality of chrominance frames in theplurality of images is used in at least two color image frames.
 8. Thesystem of claim 1, wherein the plurality of image reference framesproduced by the imaging sensor comprises: a luminance frame of longexposure pixel data and short exposure pixel data; a red chrominanceframe of long exposure pixel data and short exposure pixel data; and ablue chrominance frame of long exposure pixel data and short exposurepixel data.
 9. The system of claim 1, wherein the emitter produces apulsing pattern of luminance, chrominance blue, luminance, chrominancered.
 10. The system of claim 1, wherein the emitter produces a pulsingpattern of luminance, chrominance blue combined with chrominance red,luminance, chrominance blue combined with chrominance red.
 11. Thesystem of claim 1, wherein the emitter comprises a plurality of sourcesthat each emit a pulse of a portion of electromagnetic spectrum.
 12. Thesystem of claim 11, wherein the plurality of sources are configured tobe actuated simultaneously and wherein the plurality of sources areconfigured to produce a pulse of a predetermined interval.
 13. Thesystem of claim 12, wherein the pulse of the predetermined interval isadjusted to provide luminance information during generation of luminanceframes by matching to color space conversion coefficients.
 14. Thesystem of claim 12, wherein the pulse of the predetermined interval isadjusted to provide chrominance information during generation ofchrominance frames by matching to color space conversion coefficients.15. The system of claim 1, wherein a luminance coefficient is added tochrominance frames by an image signal processor and wherein theluminance coefficient is an integer that is a multiple of (½)^(n). 16.The system of claim 1, wherein the imaging sensor is a monochromesensor.
 17. The system of claim 1, wherein the plurality ofelectromagnetic wavelengths are emitted at differing magnitudes, whereinthe differing magnitudes correspond to the imaging sensor's sensitivityto differing wavelengths.
 18. The system of claim 1, wherein the systemfurther comprises an endoscope for accessing the ambient light deficientenvironment comprising a lumen and a hand piece; wherein the imagingsensor is disposed within one of the lumen and the hand piece.
 19. Thesystem of claim 1, wherein the system produces one color image frame pertwo captured image reference frames of the plurality of image referenceframes.
 20. The system of claim 19, wherein the emitter is pulsed andthe plurality of image reference frames are captured at a rate of 120Hz; and wherein the plurality of image reference frames are combined toproduce a plurality of color image frames; wherein the plurality ofcolor image frames are combined into a video stream having a frame rateof 60 Hz.